[Feat] Add local Cua-S1 4B and screenshot target selection - #32

What this enables

The desktop agent can use a local Cua-S1 4B model to choose a click target from the current window's screenshot. This supports controls that have no accessibility element.

The task defines named candidate points with --pixel-target. The model receives the screenshot and selects one of those points. Each click carries the window target and capture ID, so the driver can reject stale captures and invalid coordinates.

The optional cua-four-b extra supports the text and multimodal adapters through the existing decision-model interface. CUA_S1_VARIANT=4b selects it; Nano remains the default for --model cua. Screenshot bytes are passed separately from the serialized observation state.

Implements RFC #30. Related features: #14 and #15.

macOS results

Tested on macOS 26.2 arm64 with Cua Driver 0.30.1. The native app draws Save and Cancel without accessibility children and swaps their positions on reset.

The model selected right, left, right, left as Save moved. All four selections were checked against the app's output file. Median model decision time was 7.369 s. Episode times exclude model loading; the model run used no fixed plan or hosted model API calls.

An intentionally wrong-point check was rejected and recorded Cancel in the output file.

Tests

Added tests/test_decision_models_cua_four_b.py and tests/test_desktop_visual.py, and extended the model factory tests. Coverage includes option mapping, invalid model outputs, image delivery, temporary-image cleanup, capture-bound clicks, dry runs, window selection, and separate driver sessions.

Windows regression tests also cover title-based launch selection, missing or duplicate matches, and PID/window-ID binding using mocked driver responses.

Core + dev suite: 509 passed, 41 skipped. Ruff, ty, lock check, package build, shell syntax, and CLI/MCP smoke passed.

How to run

Requires macOS, uv, Swift command-line tools, and Cua Driver with Accessibility and Screen Recording permissions. Keep the Mac unlocked and close any older S1A visual fixture. Run from the current project directory:

S1A_PROJECT_DIR="$(pwd -P)"
uv sync --project "$S1A_PROJECT_DIR" --extra cua-four-b
bash "$S1A_PROJECT_DIR/evals/desktop/build_visual_fixture.sh" /tmp/S1AVisualFixture.app
export CUA_DRIVER_BIN=/Applications/CuaDriver.app/Contents/MacOS/cua-driver
export CUA_S1_VARIANT=4b CUA_S1_MODALITY=multimodal
export CUA_S1_DEVICE=mps CUA_S1_DTYPE=bfloat16 HF_DEACTIVATE_ASYNC_LOAD=1
export PYTORCH_ENABLE_MPS_FALLBACK=1
: > /tmp/s1a-visual-target.txt
uv run --project "$S1A_PROJECT_DIR" --no-sync s1a run desktop \
  --model cua --rethink off --episodes 4 --max-steps 1 \
  --app S1AVisualFixture --app-path /tmp/S1AVisualFixture.app \
  --window-title "S1A Visual Fixture" --expect Saved \
  --goal "Click the tile labelled Save in the screenshot" \
  --pixel-target left=0.27,0.51 --pixel-target right=0.73,0.51 \
  --clear Reset --execute
uv run --project "$S1A_PROJECT_DIR" --no-sync python -c \
  'from pathlib import Path; rows = Path("/tmp/s1a-visual-target.txt").read_text().splitlines(); assert rows == ["Save selected"] * 4, rows; print("Verified all four clicks")'

The base model and adapter download on first use. CUA_S1_BASE_MODEL and CUA_S1_CHECKPOINT can point to downloaded model directories.